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Abstract 


Reading proficiency continues to elude a large number of school children, in particular, our 
economically disadvantaged students (Clark & Akerman, 2006). In a continued effort to 
decrease the gap between the high-poverty and low-poverty students’ reading scores, this study 
examined the effect of third-grade and fifth-grade teachers’ use of higher-order thinking (HOT) 
question levels and student context (whole group, small group, and one-one-one) during reading 
workshop on economically disadvantaged students’ reading achievement. Teacher 
questionnaires and researcher observations were analyzed to determine the effect on reading 
CRCT scores and the two domains within the reading CRCT scores of Literary Comprehension 
and Reading Skills and Vocabulary Acquisition. In addition, intraclass correlation coefficients 
compared teacher reported use of HOT question levels to researcher observed teacher use of 
HOT questions. Results suggested the teachers were not cognizant of the higher-order thinking 
question level they used during reading instruction. Hierarchical regression analysis of the 
teachers’ use of HOT question levels suggested the questioning levels of remembering and 
understanding have a statistically significant effect on the reading CRCT scores of the 
economically disadvantaged students. However, the combined effect of all the HOT question 
levels did not statistically significantly affect the reading CRCT scores of third-grade and fifth- 
grade economically disadvantaged students. HOT questions during whole group reading 
instruction were statistically significant for the economically disadvantaged students’ scores on 
the reading CRCT and Literary Comprehension domain. In fifth-grade, student context had a 
negative correlation across each of the reading assessments with small group have the greatest 
negative correlation with reading CRCT scores. Recommendations for future research of HOT 


questions effecting economically disadvantaged students include exploring the teachers’ use of 
HOT questions in varying academic disciplines, the use of one-on-one instruction with 
economically disadvantaged students, and exploring if student context (i.e. whole group, small 
group, and one-on-one instruction) affects achievement based on the students’ grade-level. 

Keywords: higher-order thinking economically disadvantaged students, student context, teacher 
cognizance 
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Higher-Order Thinking Questions Affecting Economically 


Disadvantaged Students' Reading Achievement 

CHAPTER ONE 

Introduction 

Learning to read is one of the most important skills a student acquires throughout 
the twelve years of education. Reading influences every part of the student’s years in 
school and throughout his/her life. Learning to read opens doors of new knowledge and 
broadens a student’s world. Reading can take students to places they have not yet been 
nor may ever physically be. The ability to read can make available the career choice of 
an individual or deny their choice. However, reading proficiency continues to elude a 
large number of school children, in particular, our economically disadvantaged students 
(Clark & Akerman, 2006). 

Research indicates children from poverty have been disproportionately identified 
at risk of academic failure (Natriello, McDill, & Pallas, 1990). We cannot blame the 
student for academic failure. When research indicates a 35 point gap between the high- 
poverty and the low-poverty student reading achievement scores after years of mandates, 
reforms, and laws, we must continue to examine the teaching strategies within the 
classroom. (Aud et ah, 2010) 
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Duffy-Hester (1999) suggests the teacher is the most important factor in students’ 
learning to read. Identifying effective teaching strategies which lead to the success of 
economically disadvantaged students must be a priority of educators. Effective reading 
teachers escort learners to a lifelong love of literacy (Blair, Rupley, & Nichols, 2007). 

The Center for Improvement of Early Reading Achievement (CIERA) School 
Change study (Taylor, Pearson, Peterson, & Rodriquez, 2003) proposes higher-level 
comprehension instruction is important. The three-year study suggests higher-level 
comprehension instruction is an under-emphasized aspect of reading instruction. 
Particularly in grades two through five, the higher-level talk about text and writing are 
associated with student comprehension growth and student reading fluency. Teachers 
who ask a high proportion of higher-level questions were also identified as refraining 
from asking many low-level questions on story events and using comprehension 
questions to monitor students’ reading progress. 

The Florida Center for Reading Research conducted a study of the Florida 
Comprehensive Assessment Test (FCAT) and adolescent literacy and reading 
comprehension (Torgessen, 2004). They reviewed the reading, language, and cognitive 
abilities that are most important in explaining student performance differences on the 
FCAT at 3 rd , 7 th , and 10 th grade. The Florida Center for Reading Research examined the 
FCAT because they believe the test places high demands on vocabulary and 
reasoning/inferential skills. They found the proportion of test questions requiring higher- 
order thinking skills increases from 30% in third grade to 70% in tenth grade. 
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Consequently, teachers’ use of higher-order thinking questions must increase as students 
are promoted through elementary, middle, and high school. 

As educators face new challenges to ensure low-perfonning students achieve the 
learning standards, the value of students utilizing higher-order thinking skills to pass state 
mandated tests is manifest. Matt Cardoza, Director of Communications for the Georgia 
Department of Education, states the Georgia test items contain a range of difficulty and 
complexity. Cardoza (as reported by Downing, 2010) emphasizes Georgia tests employ a 
model called Depth of Knowledge ranging from level 1 (skill/concept) to level 4 
(extended thinking). The extended-thinking level encompasses the higher-order thinking 
skills of analyze, create, and critique. Therefore, asking students higher-order thinking 
questions will lead to students’ familiarity with the skills necessary to answer the various 
levels of test questions. 

In 2013 - 2014, Georgia state schools will begin to implement the Common Core 
State Standards Initiative proposed by the National Governors Association Center for 
Best Practice for English Language Arts and Math (National Governors Association 
Center for Best Practices, 2010). The common core standards will focus on the content 
and skills necessary to be college and career ready upon completion of high school. 

Along with providing clarity and consistency among schools, the common core standards 
will provide a focus on the application of knowledge through high-order thinking skills 
(National Governors Association for Best Practices, 2010). The higher-order thinking 
questions teachers ask students will prepare students to meet the challenges of the 
common core standards and also the challenges of competing in a global economy. 
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Purpose of the Study 

The purpose of this study is to identify the higher-order thinking question levels, 
which teachers use with students, having the greatest influence on economically 
disadvantaged students’ reading achievement. The central aim of the study is to 
determine whether particular higher-order thinking question levels increase the 
achievement of economically disadvantaged students. 

Teachers bring into their classrooms a variety of personalities, experiences, 
knowledge, and beliefs about education that influence the teaching strategies they employ 
within the school day. Although elementary school teachers are required to meet state 
and federal requirements to receive a teaching degree, each teacher enters the classroom 
having a range of knowledge and beliefs about what educating students entails. It is 
through teacher’s experiences, reflections, and professional learning that the teacher’s 
knowledge and beliefs are strengthened, challenged, refined, and implemented to meet 
the needs of all students. 

The quality of instruction students receive is a major factor in their reading 
success. Anderson, Hiebert, Scott, and Wilkinson (1985) suggest an unquestionable 
conclusion of research is the quality of teaching makes a significant difference in 
children’s learning. In fact, research demonstrates the clear correlation between student 
academic achievement and teacher quality (Whitehurst, 2002). Whitehurst (2002) 
suggests the Elementary and Secondary Education Act (ESEA) (1965) assumes teacher 
quality is affected by the teacher’s general knowledge and ability, certification and 
licensure, teaching experience, knowledge of subject matter he or she teaches, in-service 
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training that is intensive and focused, and the alignment of teacher training and standards- 
based reforms. 

High quality, effective teachers are critical to the future well-being of a child and 
to the nation’s future. Cochran-Smith (2008), a former president of the American 
Educational Research Association (AERA), states: 

There is an unprecedented emphasis on teacher quality in the United States and in 
many nations around the world. And there are extremely high expectations for 
teacher perfonnance in the twenty-first century. Based on the assumption that 
education and the economy are inextricably linked, it is now assumed that 
teachers can, and should, teach all students to world-class standards, serve as the 
linchpins in educational reforms of all kinds, and produce a well-qualified labor 
force to preserve the nation’s position in the global economy (p. 271). 

The No Child Left Behind (NCLB) Act of2001 (2002) is designed to improve 
student achievement and close achievement gaps. NCLB (2002) initiates teacher 
accountability provisions with the goal of every child achieving state-defined education 
standards in reading, language arts, math and science by the end of the 2013-2014 school 
year. Student achievement data is disaggregated by subgroups of students (e.g., race, 
gender, students with disabilities, economically disadvantaged) in order to hold schools 
accountable for making adequate yearly progress (AYP) toward the goal. Closing the 35 
point gap between children from low-poverty and high-poverty families would address 
the performance standards accountability of the subgroup economically disadvantaged 
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student population. Improving the effectiveness of teachers is critical to efforts to raise 
student achievement, narrow achievement gaps, and reduce economic disparity. 

Children represent a disproportionate share of the poor in the United States. 
Children are 25 percent of the total population, but make up 35 percent of the poor 
population (National Center for Children in Poverty, 2012). In 2010, 16.4 million 
children under age 18, or 22 percent, were poor (U. S. Bureau of the Census, 2011). The 
poverty rate for children also varies substantially by race and origin with poverty rates for 
white non-Hispanic children 11.9 percent, African-American children 35.4 percent, 
Hispanic children 33.1 percent, and Asian children 13.3 percent (U.S. Bureau of the 
Census, 2011). With the poverty rate for children expected to increase, educators must 
focus on finding successful strategies to teach all children to read and to be successful 
within the school setting and within society (Murdock, Zey, Cline, & Klineberg, 2010). 

There are a variety of reasons the economically disadvantaged students struggle 
academically beyond the control of the school system. However, it is not acceptable to 
fault the student for not learning the way teachers are currently instructing. Payne (2008) 
suggests educators can neither excuse students nor scold them for not knowing; as 
educators we must teach them and provide support, insistence, and expectations. 
Educators must examine what they are doing within the classroom to find out what will 
increase achievement for the economically disadvantaged student rather than faulting the 
student for not learning. 

This quasi-experimental, quantitative, case-study design will identify the higher- 
order thinking (HOT) question levels employed during reading instruction by analyzing 
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teacher identified use of higher-order thinking questions, researcher observation, and the 
Criterion-Referenced Competency Test (CRCT) (Georgia Department of Education, 
2012a) reading scores of economically disadvantaged students. Higher-order thinking 
usage of four 3 ld -grade and four 5 th -grade teachers within a middle Georgia elementary 
school was examined. Teacher participation was voluntary. The student scores in the 
convenience samples were from third- and fifth-grade economically disadvantaged 
students. Third-grade and fifth-grade students were chosen because the state of Georgia 
identifies these grades as “gate” years, requiring students to pass a competency test to be 
promoted to the next grade (Georgia Academic Placement and Promotion Policy, 2012). 

Data were collected and analyzed to determine if the teacher participants’ use of 
higher-order thinking questions supported economically disadvantaged students CRCT 
reading scores. Of the 94 students in third grade, 20 students meet the criteria to be 
considered economically disadvantaged. In fifth grade, the number of economically 
disadvantaged students is 26, out of a total fifth-grade enrollment of 113. 

Teacher participants in the study completed a questionnaire (Appendix A) 
identifying self-reported use of higher-order thinking questions. In addition, observations 
of teachers during reading instruction were conducted by the researcher (Appendix B). 
The teachers’ use of higher-order thinking questions was recorded to determine the level 
and frequency of each question asked during reading instruction. The researcher also 
recorded the context of students (whole group, small group, or one-on-one) during 
Reading Workshop to examine whether reading achievement was influenced by student 
context. Next, the economically disadvantaged students’ CRCT reading scores from each 


7 



teacher’s classroom were collected. The researcher examined the data to identify how the 
most frequently used higher-order thinking questions correlated to the effect on reading 
scores of the identified students. In addition, the researcher compared teacher-reported 
use of HOT question to researcher-observed teacher use to identify if teachers were 
cognizant of the HOT question asked to students. 

The data collected in this study was used to examine higher-order thinking 
questions employed during reading instruction and the potential relationship that specific 
higher-order thinking question levels had on the results of high-stakes testing for students 
in grades three and five. The results of the data analysis have been used to identify the 
higher-order thinking question levels for teachers to use to affect the reading achievement 
of students identified as economically disadvantaged. 

Definitions 

Adequate Yearly Progress - Annual targets set by states for all students (including 
identified subgroups of students) enrolled in grades 3, 4, and 5 in reading, language arts, 
math, and science with the goal of all students achieving proficiency by 2013-2014 
(Torgessen et ah, 2007). 

Economically disadvantaged students - Students determined eligible to participate in the 
Free Lunch Program under the National School Lunch Act (Governor’s Office of Student 
Achievement, 2012) 

Highly Qualified Teachers - A teacher who meets the identified requirements of having a 
Bachelor’s degree, full state teacher certification, and demonstrates competency through 
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passing a state exam or meeting standards established by each state for teacher evaluation 
(U. S. Department of Education, 2004). 

Criterion Referenced Competency Test (CRCT) - Georgia assesses students in grades 
three through eight to measure how well students acquire the skills and knowledge 
defined with the Georgia Performance Standards (GPS) in the content areas of reading, 
English/language arts, math, science and social studies. In order to meet the state’s 
minimum level of proficiency students must score 800 or above (Georgia Department of 
Education, 2007c). 

Assumptions of the Study 

Assumptions of the research included: (a) Elementary teachers at the chosen 
elementary school implement the selected county’s language arts units following the 
curriculum map and pacing guides, (b) The content of instruction, “the what”, is the 
same across grade level classrooms. However, “the how” of instruction will vary, (c) 
Teachers focus daily reading instruction, Reading Workshop, for approximately ninety 
minutes, (d) Teachers are using assessment to guide the classroom instruction to meet 
the specific strengths and/or needs of ah students, (e) Ah teachers in the study meet the 
Highly Qualified criteria as established by the state of Georgia, (f) Student performance 
on the reading CRCT is an accurate indicator of reading achievement. 
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Research Questions 

In an effort to conduct an in-depth study of the higher-order thinking questions 
affecting economically disadvantaged students, this study was driven by the following 
research questions: 

(Rl) To what extent are teachers’ cognizant of using HOT questions during 
reading instruction? 

(R2) Does the difference in teachers’ use of the levels of HOT question affect the 
reading achievement of economically disadvantaged students? 

(R3) Does student context when teachers ask higher-order thinking questions 
affect reading achievement of third-grade and fifth-grade economically 
disadvantaged students? 

Although many teaching strategies are labeled as effective, the concern of this 
research was to discover the particular higher-order thinking question levels influencing 
economically disadvantaged students. Another variable considered was to determine 
whether the student grouping (i.e. whole group, small group, or one-on-one) when the 
teacher asked the higher-order thinking questions affects student achievement. 

Therefore, this study was conducted to provide understanding into the higher-order 
thinking skills that best support economically disadvantaged students’ reading 
achievement. 
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CHAPTER TWO 

Review of Literature 

History of Educational Reform Aimed at Closing the Achievement 
Gap 

A “War on Poverty” was started in 1965 by President Lyndon Johnson with the 
Elementary and Secondary Education Act (ESEA) of 1965. Special funding (Title I) was 
earmarked for economically disadvantaged children in an effort to close this achievement 
gap. Over the forty-seven years since the ESEA Act, a multitude of initiatives have been 
funded aiming to create an equitable education for all students. 

Improving America’s School Act of 1994 (IASA) (1994) implemented a time 
frame for states to create content and performance standards and reading and math 
assessments for all students. In addition, the IASA drew attention to school wide reform 
and accelerated achievement growth as measured through adequate yearly progress 
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(AYP). IAS A also allocated funds for schools based on the poverty level of the students 
(Letendre, 1996). 

Along with funding came accountability. The Reading Excellence Act of 1998 
(1998), No Child Left Behind Act of 2001 (2002), and the Reading First and Early 
Reading First programs mandated by NCLB (2002) have increased public attention of 
student academic success leading to increased accountability of teachers and education. 
One of the key provisions of the NCLB Act (2002) is states must annually assess students 
in reading and math in grades 3-8. In addition, states must set annual targets for all 
students (including subgroups of students based on racial/ethnic groups, income levels, 
students with disabilities, and limited English proficient) with the goal of all students 
reaching proficiency in reading and math by 2013-2014 (Stullich, Eisner & McCrary, 
2007). 

However, even with all the Federal and State laws increasing funding to schools, 
increasing accountably of teachers, and using scientifically-based reading research 
(SBRR) to teach students to read, current evidence suggests that the reading achievement 
of students from low-income families has not been positively affected (Allington, 2009). 
Allington suggests in spite of large amounts of money and effort, schools have failed to 
provide struggling readers with what he believes they need most, more and better reading 
instruction. 
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Effective Reading Teachers 

In an effort to define research-based qualities of an excellent reading teacher, the 
International Reading Association (IRA) (2000) issued a statement identifying the 
following teacher qualities: (1) Teachers understand the development of reading and 
writing and possess a belief that all children can leam to be readers and writers. (2) 
Teachers continually assess student reading progress to guide reading instruction. (3) 
Teachers have a repertoire of effective teaching strategies and scaffold instruction based 
on student demonstrated abilities. (4) Teachers provide a wide variety of reading 
materials and texts for student use. (5) Teachers create flexible reading groups based on 
student’s individual strengths or needs. (6) Teachers teach reading strategies to students 
to promote independence (International Reading Association, 2000). Additionally, 
excellent reading teachers have strong content and pedagogical knowledge, manage 
classrooms to ensure a high rate of student engagement, encourage independent learning 
through using motivational strategies, hold high expectations for student achievement, 
and provide assistance to struggling students (International Reading Association, 2000). 

Blair, Rupley, and Nichols (2007) examined the position statement on Excellent 
Reading Teachers (International Reading Association, 2000) and a wide range of position 
papers to identify common instructional features associated with effective reading 
teachers. They concluded the “what” and “how” of effective reading instruction includes 
(1) administering a variety of assessment tools such as formal and informal tests, 
interviews, observations, samples of student work, portfolios, and student reflection, (2) 
explicit instruction in reading skills and strategies including modeling and guided 
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practice, (3) providing opportunities for authentic reading tasks to learn and apply skills 
and strategies based on desired learning outcomes, (4) actively engaging students by 
reading appropriate materials, (5) holding high, realistic expectations for students and 
communicating them to students in addition to high teacher self-efficacy. 

Flynn (2007) researched what effective teachers do within the classroom to affect 
student literacy in a student population in which more than 50% of the students were 
eligible for free-school meals. Flynn (2007) studied three teachers in inner-city England 
primary schools who were identified as effective based on the head teacher’s 
observations and student achievement scores. Flynn (2007) suggests effective reading 
teachers possess a range of teaching practices influenced by the behaviors of the teacher, 
teacher subject knowledge, and teacher-pupil interaction. Teachers’ modeling, 
explanations, and high-quality questioning led to students’ accumulation of reading skills 
and comprehension of reading materials. 

Topping and Ferguson (2005) studied highly effective literacy teachers to discover 
whether the teaching behaviors were consistent between teachers and different literacy 
teaching contexts, and whether teacher perceptions of their teaching behaviors 
corresponded with researchers’ observation of teacher behaviors. Researchers’ 
observations during shared reading and general literacy sessions suggested effective 
teachers most frequently used interacting with students, followed by questioning, 
transmitting information, non-teaching behaviors, and assessing. They found the 
teachers’ reported using less complex behaviors than the more complex behaviors that the 
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researcher observed. This suggests that the teachers in this research study were unaware 
of some of the behaviors they employed during reading instruction. 

Effective Teachers and Higher-Order Thinking (HOT) 

Benjamin Bloom (1956) developed a hierarchical classification of educational 
objectives that is key to understanding the learning process. According to Bloom, 
learning occurs in three domains: affective, psycho-motor and cognitive. Within the 
cognitive domain, Bloom categorizes and orders thinking skills and objectives 
representing a continuum of Low-Order Thinking Skills (LOTS) to High-Order Thinking 
Skills (HOTS). Identified as low-order are the thinking skills required to know, 
understand, and apply. Analyzing, evaluating and creating are classified as higher-order 
thinking skills. 

Bloom’s taxonomy was revised in 2001 (Anderson & Krathwohl) to exchange the 
noun descriptors of Bloom’s Taxonomy with verbs. In addition, a reordering of the high- 
order thinking skills placement of evaluation and creation moved creation to the highest 
position followed by evaluation. Anderson & Krathwohl (2001) also created descriptive 
sub-categories to outline the activities students engage in at each level of the hierarchy. 
For example, remembering is defined as retrieving, recognizing and recalling relevant 
knowledge from long-term memory (Forehand, 2010). 

Effective literacy teachers model and teach both lower-order (decoding) and 
higher-order (comprehension) skills (Pressley, Rankin and Yokoi, 1996; Knapp et al., 
1995). The effective teachers report monitoring student progress through the use of 
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questioning and students summarizing the main ideas of material read in order to promote 
meaning. Knapp et al. (1995) detennined reading teachers who stressed meaning 
creation and higher-level thinking skills (in addition to lower-level skills) were more 
effective in increasing student achievement than others. 

In another study focusing on effective teaching practices, Wray, Medwell, Fox 
and Poulson (2000) compared the teaching practices of identified “effective literacy 
teachers” to teachers that were not identified as “effective literacy teachers”. The 
teachers not identified as “effective literacy teachers” failed to demonstrate to the same 
degree the specific teaching practices employed by the effective teachers. The teaching 
practices most often identified among the effective teachers included (a) using shared text 
to teach a range of literacy skills and knowledge at the word, sentence and text level, (b) 
providing a distinct beginning and end to each literacy teaching session, (c) employing a 
brisk pace with limited time frames for sub-tasks, (d) refocusing student attention 
regularly (e) teaching the purpose and processes of literacy explicitly, (f) modeling and 
demonstrating reading skills and strategies, (g) using a wide range of open-ended 
questions about reading decisions and strategies. 

Bitter, O’Day, Gubbins, and Socias (2009), in a three year study of San Diego 
City Schools’ (SDCS) instructional practices in a balanced literacy approach, found the 
teachers’ use of high-level questions and discussion of text were associated with students’ 
growth in reading achievement. However, when they replicated the 2004-2005 study in 
2005-2006, they did not observe the same significant positive effects of high-level 
comprehension instruction. Upon reanalysis of the data to explore possible reasons for 
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discrepancies, they discovered an increase in the number of English Learners (EL). Thus 
they suggest that differential effectiveness of literacy practices for different groups of 
students is an area for further research. 

Taylor, Pearson, Peterson, & Rodriguez (2003) suggest that schools have a 
plethora of information available to them to improve student reading. However, they 
suggest that often the dilemma is knowing where to focus their efforts due to so many 
options. Taylor et al. (2003) recommend teachers focus on elements of reading 
instruction that maximize students’ cognitive engagement. Their study suggests higher- 
level thinking and application of reading strategies have the greatest impact on student 
reading achievement. The “how” (the teaching process) is of vital importance in student 
reading success. 

McNeil (2010) researched the teacher’s use of HOT questions and English 
Language Learners (ELL). Through teacher observation, teacher interviews, and student 
questionnaires, he suggests the frequency and type of HOT questions teachers use impact 
ELL students in addition to the perceptions, attitudes, and beliefs of the students. McNeil 
(2010) also suggests the context in which teachers ask HOT questions and the teacher’s 
reason for asking the HOT questions are areas for further research. 

Economically Disadvantaged Children 

Although increased attention has focused on literacy achievement across 
socioeconomic lines, the gap remains. The National Assessment of Educational Progress 
(NAEP, 2005) estimates that approximately 37% of fourth grade students failed to 
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achieve at the basic reading level. The number of students scoring at this lowest level has 
remained relatively constant with only minimal gains over many years (NAEP, 2005) On 
each NAEP assessment given to fourth-grade students from high-poverty schools 
between 1998 and 2009, the reading scores for the high-poverty students were lower than 
student reading scores from low-poverty schools (Kerachsky, 2010). 

National Assessment of Educational Progress (NAEP, 2011) reports the average 
2009 reading score for a fourth grade student in Georgia was 221 (National average 
scores were 221). Sixty-six percent of students scored at or above Basic (partial mastery 
of fundamental skills), while thirty-two percent of students scored at or below proficiency 
levels (demonstrated competency over challenging subject matter). 

Twenty-four percent of all public elementary schools in the South in 2007-2008 
met the qualifications to be titled a high-poverty school (Aud, et al. 2010). High-poverty 
schools have 76-100 percent of students eligible for free/reduced-price meals. 

Nationally, approximately 6 million students attend a high-poverty school. Nationwide, 
students attending schools with a free/reduced-price lunch population of 0-25 percent had 
an average fourth-grade reading score of 238. Students attending schools with a 
free/reduced-price lunch population of 26-50 percent scored 226 on the reading 
assessment. Students attending schools with a free/reduced-price lunch population of 51- 
75 percent had an average reading score of 217. A free/reduced-price lunch student 
population of 76-100 percent had an average reading score of 203 (Aud et ah, 2012). 

This is a 35 point gap between high-poverty students’ fourth-grade reading scores when 
compared to low-poverty student reading scores. 
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In high-poverty schools, Hispanics and Blacks represent the greatest share of 
enrollment at both the elementary and secondary level (Kerachsky, 2010). In 2007-2008, 
almost 25 percent of students attending high-poverty elementary schools were identified 
as Limited-English Proficient (LEP) students (U.S. Department of Education, 2008). 
These figures have not changed significantly in the past fifteen years. In the Review of 
the Condition of Education, Young (1997) states: 

of the 4 million babies born each year, almost one out of three is bom to a mother 
who lives in poverty.. .these conditions have been shown to be associated with 
children experiencing problems such as repeating a grade, requiring special 
education services...students from racial/ethnic minority backgrounds and low 
income families are more at risk for poor school outcomes and are becoming an 
increasing share of the student population (p. 3). 

School enrollment in the United States is expected to continue to grow. From 
2007 through 2019, public elementary and secondary education enrollment is projected to 
increase to 52 million students. The southern states are expected to experience the largest 
increase in the number of students enrolled from 2007-2009 to 2019-2020 to an 
enrollment of 40 percent of the total population of students in school (Kerachsky, 2010). 

The National Center for Children in Poverty (2012) reports that nearly 15 million 
children in the United States - 21% of all children - live in families with incomes below 
the federal poverty level ($22,350 a year for a family of four). Although most parents of 
these children work, the low wages and economic instability keep these families within 
the federal poverty level. Focusing on children between the ages of six and eleven, 44% 
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of children live in low-income families while 21 percent live in poor families for a total 
of more than 24 million children (National Center for Children in Poverty, 2012). 

Poverty has a wide range of effects for children. The National Center for Children 
in Poverty (2012) suggests poverty can contribute to poor health and mental health 
problems. Children who experience poverty when they are young and/or experience deep 
and persistent poverty are at the greatest risk for academic, social, emotional or 
behavioral problems (National Center for Children in Poverty, 2012). Research is clear 
that poverty is the single greatest threat to children’s well-being (Payne, 2008). 

Effects of Poverty on Children 

The effects of poverty on children vary depending on a variety of differential 
factors such as timing, depth, and duration of childhood poverty (Duncan, Yeung, 
Brooks-Gunn, & Smith, 1998). Brooks-Gunn and Duncan (1997) report negative 
outcomes within the physical health, cognitive, school achievement, and emotion or 
behavior aspects of children of poverty. They report children living below the threshold 
of poverty are 1.3 times as likely as non-poor children to suffer learning disabilities and 
developmental delays. 

Noble, Wolmetz, Ochs, Farah and McCandliss (2006) researched the relationship 
between cognitive skill and reading-related brain activity of students in low- 
socioeconomic versus high-socioeconomic environments. Through the use of functional 
magnetic resonance imaging (fMRl), researchers suggest that a child’s experiences and 
background, operationalized by socioeconomic status (SES), influence the brain activity 
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in the left fusiform gyrus. This area in the brain has been associated with the visual- 
orthographic process in reading (Brunswick, McDrory, Price, Frith, & Frith, 1999; Fiez, 
Baloto, Raichle, & Petersen, 1999; McCandliss, Cohen, & Dehaene, 2003; Price & 
Devlin, 2004) and is positively associated with reading skill deficits in children 
(Shaywitz et ah, 2002). This study suggests the lack of literacy resources in low-SES 
environments affects students’ development of this area of the brain which controls 
acquiring the phonological skills necessary to read. 

In another study, Noble, Farah and McCandliss (2006) researched how 
socioeconomic background modifies cognition-achievement in reading. They examined 
whether the variance in reading ability, accounted for by phonemic awareness (PA) and 
socioeconomic status (SES), interact where previous studies have only examined these 
factors separately. They suggest children from lower socioeconomic backgrounds are 
consistently at a disadvantage when examining reading achievement. 

Hart and Risley (1995) conducted a two and one-half year study focusing on the 
rates of vocabulary growth of children. They compared the vocabulary used within the 
homes of professional families, working-class families, and families receiving welfare. 
They found that the amount parents speak to their infants and toddlers is generally 
correlated with the parents’ economic status. Hart and Risley (1995) suggest children 
from welfare recipients’ homes hear fewer words per hour (616 words compared to 2153 
words), hear less of a variety of vocabulary, and the use and quality of words heard is 
fewer. They suggest the lack of exposure to a strong vocabulary is correlated to 
standardized test measures of the child’s intellectual achievement at age three (r = .78) 
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and then at age nine (r = .77). Biemiller (2007) suggests the lack of vocabulary is the 
main reason students have difficulty understanding text. 

Why is HOT Important? 

Will students be ready to face the challenges of the 21 st century with the education 
they are receiving today? Society is changing at a rapid pace. Sir Ken Robinson, a 
creativity expert, states we are living in a day of “massive unpredictability” (Azzam, 
2009) yet educators are charged with the task of preparing children for a world of 
unknowns. The ability to use infonnation in critical thinking and problem solving in 
order to increase students’ adaptability to the changing world is vital to students’ future 
(Beachboard & Beachboard, 2010). Higher-order thinking questions allow students the 
opportunity to acquire the cognitive passport of the future (Martinez, 1998). 
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CHAPTER THREE 


Method 


Introduction 

The purpose of this quasi-experimental study was to determine if teachers’ use of 
higher-order thinking (HOT) question levels affects the reading achievement of 
economically disadvantaged students in third and fifth grade. Another variable to be 
researched is to review students’ context (defined as whole group, small group or one-on- 
one) when teachers ask the higher-order thinking (HOT) questions to determine if context 
influences economically disadvantaged students’ reading achievement scores. A 
comparison of teacher-reported use of HOT questions to researcher’s observation of HOT 
questions will also be examined. 

Rationale for Research Design 

According to Creswell (2008), “If the problem calls for (a) the identification of 
factors that influence an outcome, (b) the utility of an intervention, or (c) understanding 
the best of outcomes, then a quantitative approach is best” (p. 18). Creswell (2008) 
suggests quantitative design is compatible when the researcher is examining the 
relationship between variables and posing questions using surveys or questionnaires to 
collect information to produce statistical data. The descriptive data collected allowed the 
researcher to identify current teachers’ beliefs of their use of higher-order thinking 
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questions, quantify the HOT question levels that teachers ask, and identify the HOT 
question level having the most positive influence on economically disadvantaged 
students’ reading achievement. 

In addition, the researcher’s observation of the context of students when teachers 
ask HOT questions was examined to explore the impact of context on economically 
disadvantaged students’ reading achievement. Student context was defined as whole 
group, small group, or one-on-one instruction. Whole group was defined as the teacher 
asking the question to the entire class and then calling upon one or more students to 
provide the answer to the question. Small group was defined as the teacher working with 
a group of six or fewer students and asking the question to the students within the group. 
One-on-one context is defined as the teacher working with only one student and the 
teacher asks only that student the question. 

A comparison of teacher-reported use of HOT questions and researcher-observed 
teacher use of HOT questions was initiated to explore if teachers have a reasonable 
estimate of how frequently they use HOT questions. This data will be valuable to 
understand if there is a significant difference between what teachers perceive they are 
doing in the classroom and what actually occurs. This information may lead to teachers 
to make a more conscious effort to use the HOT questions within their classrooms to 
affect student achievement. 
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Research Site 


The population included in this study was third- and fifth-grade regular education 
elementary teachers from a rural elementary school in the southeast. For the school year 
2011-2012, the school’s total student enrollment was 668 students (pre-kindergarten 
through fifth grade) consisting of 95 African-American students, 469 Caucasian students, 
52 Hispanic students, 2 American Indian students, 16 Asian students and 34 mixed 
students. Twenty-two percent of students were eligible for free/reduced-price meals 
(101 students eligible for free meals; 54 students eligible for reduced-price meals). One 
hundred and six third graders’ 2010-2011 CRCT reading scores indicated 97.2 % met or 
exceeded the standards with a mean score of 863.8 (Georgia Department of Education, 
2011b). One hundred forty-five fifth graders’ 2010-2011 CRCT reading scores indicated 
96.6% of students met or exceeded the standard with a mean score of 847.35 (Georgia 
Department of Education, 2011b). The 2010-2011 School Report Card (Georgia 
Department of Education, 2011a) reported 100% of the economically disadvantaged 
students in third and fifth grade met or exceeded the standards on the reading portion of 
the CRCT. 

Although all third- and fifth-grade students were included in the researcher’s 
classroom observations, the student participants reported within this study were the third- 
and fifth-grade students meeting the federal requirements to qualify for free/reduced- 
price meals. The total number of students in third grade is 94. Twenty students in third 
grade were selected as participants as they met the researchers’ identified population 
criteria as eligible for free/ reduced-price meals. In fifth grade, twenty-six students of the 
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113 students met the requirements of free/reduced-priced meals eligibility. The teacher 
participants in the study were 4 third-grade and 4 fifth-grade teachers at the elementary 
school. 

The convenience sample of third-grade and fifth-grade teachers was chosen 
because third- and fifth-grade students are required by Georgia state law to pass the 
reading portion of the CRCT in order to be promoted to the next grade level (Georgia 
Academic Placement and Promotion Policy, 2012). Permission to conduct the research 
and collect the necessary data for this study was requested from the elementary school 
principal (Appendix D) and the Office of Professional Development (Appendix E). The 
researcher requested access to the third- and fifth-grade CRCT reading scores as provided 
by the Georgia Department of Education to the school. The database provided included 
grade, teacher name, student name, student date of birth, the scale score/perfonnance 
level of the reading CRCT, the students’ Lexile score (Stenner & Smith, 1982) and the 
number of correct student responses by content domain. Names of students meeting the 
eligibility requirements of economically disadvantaged were requested from the 
elementary school in order to identify the specific reading scores of each of the eligible 
students. 

Participants 

3 ld grade teacher participants. In this elementary school, the gender of 3 third- 
grade teachers was female with only one of the 4 third-grade teachers being a male. All 
the third-grade teachers were Caucasian. The mean age of teachers was M = 44.63 (SD = 
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15.4). Years of teaching experience of the third-grade teachers was M= 16.5 years (SD 
= 10.2) (Table 1). The mean years of the third-grade teachers teaching third-grade 
students was M= 9.5 years (SD = 10.11). Two third-grade teachers had Master’s degree 
while one teacher holds a Bachelor’s of Education Degree and one teacher holds a 
Specialist of Education degree. All the third-grade teachers have clear-renewable 
teaching certificates. 


Table 1 

Descriptive Statistics of Third-grade Teacher Participant (n = 4) 


Characteristics 

Mean 

SD 

Age 

44.63 

15.40 

Years of teaching experience 

16.50 

10.12 

Years teaching 3 rd grade 

9.50 

10.11 


The self-reporting teacher questionnaire asked teachers to identify the in-service 
or professional learning they had attended within the previous two years (Table 2) as the 
professional learning of teachers is mandated under the NCLB Act of 2001 (2002). All 
third-grade teachers reported having attended in-service or professional learning 
surrounding the topic of best-teaching practices in the area of higher-order thinking for 
five or more hours. The only topic of professional learning three teachers had not 
participated in at all was focused on strategies for teaching reading to students from 
diverse backgrounds. Teachers also reported their consideration of the Bloom’s 
Taxonomy question level (Anderson & Krathwahl, 2001) before asking students 
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questions. One teacher reported she considered Bloom’s Taxonomy a small extent, one 
teacher reported consideration of Bloom’s Taxonomy a moderate extent, while two 
teachers reported they largely considered Bloom’s Taxonomy before asking students a 
question. 

In response to question number two of the teacher questionnaire (Appendix A), 
“Have you changed your methods of teaching reading in the past two years?”, half of the 
teachers indicated they had made no changes to their methods of teaching reading. 
However, two teachers reported changing their reading teaching methods citing using 
partnerships more frequently and incorporating more conferencing with students during 
reading workshop. 


Table 2 

Third-Grade Teachers Hours Spent in In-Service or Professional Learning (n = 4) 
Variable Number of hours Percentage of teachers 


How students learn to read 

Not at all (0) 



1-2 



3-4 

100% 


5+ 


Content standards in reading 

Not at all (0) 



1-2 



3-4 



5+ 

100% 

Instructional methods for 

Not at all (0) 


teaching reading 

1-2 



3-4 

100% 


5+ 


Strategies for teaching reading 

Not at all (0) 

75% 

to students from diverse 

1-2 


backgrounds 

3-4 

25% 


5+ 


Best teaching practices in the 

Not at all (0) 


area of HOT 

1-2 



3-4 
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Table 2 


5+ 


100 % 


5 th grade teacher participants. In this elementary school, the gender of all the 
fifth-grade teachers was female. Three of the fifth-grade teachers were Caucasian and 
one of the fifth-grade teachers was African-American. The mean age of teachers was M 
= 43.54 (SD = 4.9) (Table 3). Years of teaching experience of the fifth-grade teachers 
was M= 14.25 years (SD = 4.35). The mean years of teaching fifth-grade students was 
M= 7.5 years (SD = 4.65). Two fifth-grade teachers had Master’s degree while one 
teacher holds a Bachelor’s of Education Degree and one teacher holds a Specialist of 
Education degree. All the fifth-grade teachers have clear-renewable teaching certificates. 


Table 3 


Descriptive Statistics of Fifth-Grade Teachers (n = 4) 


Characteristics 

Mean 

SD 

Age 

43.54 

4.90 

Years of teaching experience 

14.25 

4.35 

Years teaching 5 th grade 

7.50 

4.65 


Teachers reported the in-service or professional learning attended within the 
previous two years (Table 4). Fifty percent of the teachers reported little (1-2 hours) in- 
service or professional development on the topic of best-teaching practices in the area of 
higher-order thinking. One teacher reported attending three - four hours while one 
teacher reported five plus hours of in-service or professional learning. Only one teacher 
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reported having no in-service or professional learning surrounding how students learn to 
read. 

Teachers reflecting on their consideration of Bloom’s Taxonomy question level 
(Anderson & Krathwahl, 2001) before asking students questions (teacher questionnaire 
number 5, Appendix A) indicated half of the teachers consider the Bloom’s taxonomy 
level (Anderson & Krathwahl, 2001) to a small extent while half of the teachers consider 
the Bloom’s taxonomy level moderately before questioning students. All fifth-grade 
teachers reported changing the methods of teaching reading in the past two years (teacher 
questionnaire number two, Appendix A). These changes include the addition of rigor and 
strategies, more read-aloud of texts, more higher-order thinking questions, asking student 
to visualize pictures for stories with the purpose of focusing on details within the text, 
conferencing with students more frequently, and more teacher reflection on teaching 
practices. 


Table 4 

Fifth-Grade Teachers Hours Spent in In-Service or Professional Learning (n = 4) 


Variables 

Number of hours 

Percentage of 
teachers 

How students learn to read 

Not at all (0) 

25% 


1-2 

25% 


3-4 

50% 


5 + 


Content standards in reading 

Not at all (0) 



1-2 

50% 


3-4 

25% 


5 + 

25% 

Instructional methods for teaching 

Not at all (0) 


reading 

1-2 

50% 


3-4 

25% 


5 + 

25% 
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Table 4 


Strategies for teaching reading to students 
from diverse backgrounds 

Not at all (0) 

1-2 

3-4 

5 + 

25% 

50% 

25% 

Best teaching practices in the area of 

Not at all (0) 


HOT 

1-2 

50% 


3-4 

25% 


5 + 

25% 


3 ld grade student participants. Students in grade three ranged in age from 

eight-years old to nine-years old at the time of CRCT administration. The mean age of 

the economically disadvantaged students was 9 years, 2 months (SD = .62). Twenty 

students met the qualifications for economically disadvantaged in third grade (twelve 

males and eight females). The mean CRCT reading score for the economically 

disadvantaged third-grade students was 849.40 (SD = 27.29). Two economically 

disadvantaged students were receiving Early Intervention Program (EIP) (Georgia 

Department of Education, 2012b) assistance for reading and two economically 

disadvantaged students were participating in Early Intervention Program (EIP) support 

for math. EIP is a state funded program which provides supplementary instructional 

resources such as an additional teacher within the classroom during reading workshop 

instruction. The EIP Program targets students performing below grade level as measured 

by a state-created rubric or failing the CRCT assessment (Georgia Department of 

Education, 2012c). Additionally, three third-grade economically disadvantaged students 

were receiving instructional support at the Tier 3 level of the Response to Intervention: 

Pyramid of Intervention (Georgia Department of Education, 2012d). Tier 3 instructional 
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support allows for students to receive intensive, research-based interventions tailored to 
the specific skill deficit of the student (Georgia Department of Education, 2012d). None 
of the third-grade economically disadvantaged students had been retained in previous 
years or were in the gifted and talented program. 

Table 5 reports the mean 2011-2012 CRCT reading score of economically 
disadvantaged students to the non-economically disadvantaged. Although less than the 
35 point standardized reading score gap between economically and non-economically 
disadvantaged students reported by National Assessment of Educational Progress (2005), 
the third-grade economically- disadvantaged students scored almost 18 points lower on 
the reading CRCT than the non-economically disadvantaged students. Reviewing the 
domain score of Literary Comprehension and Reading Skills and Vocabulary, again the 
economically disadvantaged students scored lower than their non-economically 
disadvantaged peers. 


Table 5 

Descriptive Statistic of2011-2012 Reading CRCT Scores and the Domain Scores of 
Third-Grade Students ('n = 20) = 75) _ 

Reading Skills and 

Student Literary Vocabulary 

population CRCT Reading Comprehension Acquisition 

Economically 
disadvantaged 
students 3 


M= 849.40 M= 18.8 M=6.75 

SD = 27.29 SD= 3.5 SD= .85 
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Student 

population 

Non-economically 

disadvantaged 

students’ 3 


CRCT Reading 

M= 867.37 
SD = 26.95 


Literary 

Comprehension 

M = 20.75 
SD= 3.15 


Reading Skills and 
Vocabulary 
Acquisition 

M= 7.32 
SD= .82 


Note. The CRCT reading score is a scaled score comprised of the three domains of 
Literary Comprehension, Reading Skills and Vocabulary, and Reading for Infonnation 
(Georgia Department of Education, 2012). The Literary Comprehension domain 
consists of 24 multiple-choice questions. The Reading Skills and Vocabulary Acquisition 
domain consists of 8 multiple-choice questions. 


Table 6 compares the percentage of economically disadvantaged students and 
non-economically disadvantaged students scoring at each of the perfonnance levels. 
Seventy-nine percent of the non-economically disadvantaged students scored in the 
exceed range compared to 45 % of the economically disadvantaged students scoring in 
the exceed range. Although both the groups of students had a 100% passing rate, the 
non-economically disadvantaged student passed with higher scores. 
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Table 6 


Third-Grade Student Performance Levels on Reading CRCT 2011-2012 fn = 20) A'V = 


75f 


Student 

Exceeds 

Meets 

Does not meet 

population 




Economically 

disadvantaged 

students 3 

45% 

55% 

0% 

Non-economically 

disadvantaged 

students' 3 

79% 

21% 

0% 


Note. Student scores at or above 850 indicate a performance level that exceeds the 


standard for the test. Student scores from 800 - 849 indicate performance that meets the 
standard for the test. Scores below 800 indicate perfonnance that does not meet the 
state’s minimal level of proficiency. Georgia Department of Education. (2012e). 2012 
Score Interpretation Guide: Grades 3-8. Retrieved from http://www.doe.kl2.ga.us/ 


5 th grade student participants. In fifth grade, students ranged in age from 
eleven- to twelve-years old when tested. The mean age of the twenty-six students was 11 
years, 10 months (SD = .65). Fourteen male students and twelve female students met the 
qualification of economically disadvantaged students. The mean CRCT reading score for 
the economically disadvantaged fifth-grade students was 835.69 (SD = 16.36). Two of 
the fifth-grade economically disadvantaged students were receiving EIP math services 
however, none of the students were receiving EIP reading services (Georgia Department 
of Education, 2012c). One economically disadvantaged student received special testing 
accommodations of small group, extended time, and frequent breaks based on the student 
meeting the qualifications of the Rehabilitation Act of 1973 (1973) and the Americans 
with Disabilities Act (1991) (Advocacy Incorporated, 2012). None of the fifth-grade 
economically disadvantaged students had been retained in previous years or were in the 
gifted and talented program. 
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Table 7 compares the mean reading assessment scores of economically 
disadvantaged students to the non-economically disadvantaged students. While less than 
the national 35 point difference (Aud et ah, 2012), the fifth-grade economically 
disadvantaged students scored almost 16 points lower than the non-economically 
disadvantaged students. Reviewing the domain score of Literary Comprehension and 
Reading Skills and Vocabulary, again the economically disadvantaged students scored 
lower than their non-economically disadvantaged classmates. 


Table 7 


Descriptive Statistics of2011-2012 Reading CRCT Scores and the Domain Scores of 
Fifth-Grade Students C'n = 25) /’V = 88) 


Student 

Population 

CRCT Reading 

Literary 

Comprehension 

domain 

Reading Skills and 
Vocabulary 
Acquisition domain 

Economically 

disadvantaged 

M= 835.69 

M= 10.96 

M= 7.00 

students 3 

SD= 16.36 

SD= 2.17 

SD = 1.35 

Non-economically 

disadvantaged 

M= 851.65 

M = 12.59 

M= 7.16 

students b 

SD= 26.83 

SD= 2.57 

SD = .97 


Note. The CRCT reading score is a scaled score comprised of the three domains of 
Literary Comprehension, Reading Skills and Vocabulary, and Infonnation and Media 
Literacy (Georgia Department of Education, 2012a). The Literary Comprehension 
domain consists of 16 multiple-choice questions. The Reading Skills and Vocabulary 
Acquisition domain consists of 8 multiple-choice questions. 


Table 8 compares the percentage of economically disadvantaged students and 
non-economically disadvantaged students scoring at each of the performance levels. Fifty 
percent of the non-economically disadvantaged students scored in the exceeds range 
while 24% of the economically disadvantaged students scored in the exceeds range. 
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Although both the groups of students had almost a 100 percent passing rate, the non- 
economically disadvantaged students passed with higher scores. 


Table 8 


Fifth-Grade Student Performance Levels on Reading CRCT 2011-2012 (°n = 25) = 

88 ) 


Student 




Population 

Exceeds 

Meets 

Does not meet 

Economically 

disadvantaged 

students 3 

24% 

76% 

0% 

Non-economically 

disadvantaged 

students' 3 

50% 

49% 

1% 


Note. Student scores at or above 850 indicate a performance level that exceeds the 


standard for the test. Student scores from 800 - 849 indicate performance that meets the 
standard for the test. Scores below 800 indicate perfonnance that does not meet the 
state’s minimal level of proficiency. Georgia Department of Education. (2012). 2012 
Score Interpretation Guide: Grades 3-8. Retrieved from http://www.doe.kl2.ga.us/ 


The fifth-grade students’ results from the 2010-2011 and the 2011-2012 reading 
CRCT score, Table 9, report both the economically disadvantaged students and the non- 
economically disadvantaged students have the same percentage of students meeting or 
exceeding the minimum score of 800. In 2010-2011 and 2011-2012, both groups of 
economically disadvantaged students had a 100 % met and/or exceed score. Review of 
the non-economically disadvantaged students’ 2011 and 2012 reading CRCT scores 
shows a minimal difference of only one percent in the does not meet category during the 
2012 reading CRCT administration. The percentage of economically disadvantaged 
students that exceed the standards in 2010-2011 was 44 percentage points compared with 
the economically disadvantaged students in 2011-2012 that exceeded the standard by 24 
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percentage points (20 percentage points lower than the previous year) even though both 
of the tests contained HOT questions and were created to measure the Georgia 
Performance Standards. 


Table 9 


Fifth-Grade Student Performance Levels on Reading CRCT 2010-2011 and 2011-2012 
fn = 25) fN = 120) 



Exceeds 


Meets 

Does not meet 

Student 

2010- 

2011- 

2010- 

2011- 

2010- 

2011- 

Population 

2011 

2012 

2011 

2012 

2011 

2012 

Economically 

disadvantaged 

students 3 

44% 

24% 

56% 

76% 

0% 

0% 

Non-economically 

disadvantaged 

students' 3 

44% 

50% 

56% 

49% 

0% 

1% 


Note. Student scores at or above 850 indicate a performance level that exceeds the 
standard for the test. Student scores from 800 - 849 indicate perfonnance that meets the 
standard for the test. Scores below 800 indicate perfonnance that does not meet the 
state’s minimal level of proficiency. Governor’s Office of Student Achievement. (2011). 
2010-2011 Report Card. Retrieved from http://reportcard201 Lgaosa.org/ 


Instruments 

Teacher Questionnaire. The teacher questionnaire used was adapted from 
the International Association for the Evaluation of Educational Achievement Progress in 
International Literacy Study in Primary School in 40 Countries (PIRLS) (Mullis, Martin, 
Gonzalez, Kennedy, & Foy, 2007) (Appendix A). The Cronbach’s alpha reliability 
coefficient for overall reading was 0.88. The PIRLS (Mullis et ah, 2007) teacher 
questionnaire was designed to gather data with the purpose of developing students’ 
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reading skills and strategies. The researcher selected questions from the PIRLS (Mullis 
et al., 2007) teacher questionnaire to construct the questionnaire employed. The 
following question numbers were identified as meeting the purpose of the study and 
included within the questionnaire: 11, 15, 16, 17, 29, 30, 31, 32, 33, 34, and 36. The 
researcher’s adapted version was field tested at a local elementary school before actual 
implementation at the chosen elementary school. Reliability of the researcher’s field- 
tested questionnaire was established using test-retest reliability. 

Observation Tool. The researcher’s observation tool listed each of the 
higher-order thinking skills from revised Bloom’s Taxonomy (Anderson & Krathwohl, 
2001). A category for “context” to record student grouping structure (whole group, small 
group or one-on-one) was included to infonn the researcher of students’ context when 
HOT questions were asked (Appendix B). The observation tool also allowed the 
researcher to identify the emphasis of the reading workshop lesson observed (i.e. mini¬ 
lesson, conferences or strategy groups, or teaching share). In addition, the students’ first 
name answering the teacher’s question and if the student’s answer was correct or 
incorrect was recorded. 

Reading CRCT Scores. Student reading achievement was measured using 
the results of the 2011-2012 reading CRCT administered in April 2012. The fifty- 
question multiple-choice test was administered in one day, which included a time limit of 
90 minutes for Part I and an additional 90 minutes for Part II. This standardized test was 
administered by the students’ general-education teacher, unless an Individualized 
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Education Plan (IEP) states specific accommodations for the student such as small-group 
setting. 

The dependent variables for the research were the students’ reading CRCT scores 
and the domain scores of Literary Comprehension and Reading Skills/Vocabulary 
included within the CRCT reading assessment. Students scoring below 800 on the 
reading CRCT are categorized as “does not meet” the standard. Student scores of 800 - 
849 are in the “meets” the standard category. In order to “exceed” the standard on the 
reading portion, student scores above 850 are required (Georgia Department of 
Education, 2012a). . 

The Georgia reading CRCT exam consists of three domains. In third grade, the 
domains are Literary Comprehension, Reading for Information, and Reading Skills and 
Vocabulary Acquisition. In fifth grade, the domains contained with the reading CRCT 
are Literary Comprehension, Infonnation and Media Literacy, and Reading Skills and 
Vocabulary Acquisition (Georgia Department of Education, 2012a). For this research, 
the domains of Literary Comprehension and Reading Skills and Vocabulary Acquisition 
were used for the analysis as these two domains were common within the third- and fifth- 
grade reading CRCT. 

According to the Georgia Department of Education (2007a) the Literary 
Comprehension domain in third grade includes the skills necessary to “comprehend and 
explore literary works by identifying and analyzing elements of various texts including 
short story, fairy tale, fable, folktale and poetry” (p. 9). The Reading Skills and 
Vocabulary Acquisition domain in third grade encompasses the skills necessary to “read, 
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interpret and apply difficult text and new vocabulary in a variety of texts” (Georgia 
Department of Education, 2007a, p. 8). 

The Literary Comprehension domain in fifth grade includes the skills necessary to 
“comprehend and explore literary works by identifying and analyzing elements of various 
texts including short story, dramas, folktales, poetry and descriptive narratives” (Georgia 
Department of Education, 2007a, p. 15). The Reading Skills and Vocabulary 
Acquisition domain in fifth grade “refers to the skills required to read, interpret, and 
apply difficult text and new vocabulary in a variety of texts” (Georgia Department of 
Education, 2007a, p. 14). 

In a release to the Atlanta Journal Constitution, the Georgia Department of 
Education described how the CRCT’s validity is maintained. The DOE spokesman, Matt 
Cardoza, states establishing validity for the Georgia CRCT starts with the purpose of the 
assessment and follows through item writing and review. All CRCT items were written 
by qualified-professional content specialists specifically for the Georgia CRCT. The test 
items were evaluated by curriculum specialists for overall quality and clarity, content 
coverage and appropriateness, alignment to the state curriculum, and grade appropriate 
standards. The multiple-choice test had one clear, correct answer with appropriate, 
relevant, and reasonable distracters. In addition, the Georgia Testing Division members 
met quarterly with an independent panel of experts, Georgia’s Technical Advisory 
Committee (TAC), who reviewed all aspects of the test development and implementation 
process (as cited in Downing, 2012). 
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Data Collection Method 


Participants included in this study were third- and fifth-grade general education 
teachers in a rural-southeastern elementary school. The researcher explained the purpose 
of the research and requested teachers’ participation in the research project at a before¬ 
school meeting. The researcher answered any teacher questions about participation in the 
study and then left the meeting. The request to participate (Appendix C) and the teacher 
questionnaire (Appendix A) were delivered to participants at this time by a teacher 
volunteer with instructions to return the request to participate and the completed 
questionnaire to the teacher volunteer in the provided sealable envelope within two days. 
The approximate time to complete the request to participate and the teacher questionnaire 
was ten minutes. Permission to ask the teachers to participate in the research project was 
granted by the school principal (Appendix D) and the school district (Appendix E). 

Classroom observations occurred during teacher identified reading workshop time 
as required by the district-adopted language arts curriculum for each grade level. The 
literacy curriculum for the district includes a balanced-literacy approach emphasizing the 
Georgia Performance Standards for Reading. The components of reading featured during 
Reading Workshop include reading aloud to children, shared reading with the whole 
class, guided reading with a small group of students, and students reading independently. 
Each classroom teacher was observed two times during two consecutive calendar weeks. 
A total of sixteen classroom observations were conducted with the observation times 
ranging from 30 - 35 minutes each. The teachers’ questions and student context were 
recorded by the researcher. Later the researcher assigned the questions to the appropriate 
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level of Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001). Each of the HOT 
question levels were then used as a grouping variable to identify if the HOT question 
level affected economically disadvantaged students’ reading achievement. 


Analysis Methods 

The Statistical Package for the Social Sciences (SPSS, Version 20) software was 
used to analyze the teacher questionnaire, the observation data, and the CRCT reading 
scores. 

An intraclass correlation coefficient was computed to assess the relationship 
between the teacher-reported use of HOT questions and the researcher-observed teacher 
use of HOT. Intraclass correlation coefficient is recommended when the reliability or 
consistency of repeated measures is examined. In this research, the consistency of 
teacher reported data and researcher observed data was measured using the two-way 
random model, absolute agreement, average measure intraclass correlational model 
(Shrout & Fleiss, 1979). 

Hierarchical regression analysis was conducted to examine the incremental 
contribution of the various levels of HOT questions on the CRCT reading scores of 
economically disadvantaged students in third and fifth grade. Petrocelli (2003) states the 
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“focus of hierarchical regression is the change in predictability associated with predictor 
variables entered later in the analysis over and above that contributed by predictor 
variables entered earlier in the analysis” (p 11). Hierarchical regression allows the 
researcher to “test theoretical assumptions and examine the influence of several predictor 
variables in a sequential way, such that relative importance of a predictor may be judged 
on the basis of how much it adds to the predictors of a criterion” (p 10). Based upon the 
revised Bloom’s Taxonomy theory (Anderson & Krathwohl, 2001), the HOT question 
levels were entered in the following order: remembering, understanding, applying, 
analyzing, evaluating, and creating. 

To answer the third research question, does student context when teachers ask 
HOT questions affect reading CRCT scores of economically disadvantaged students, the 
Pearson product-moment correlational analysis was implemented (Field, 2005). 
Correlational analysis best met the researcher’s interest to discover if there was a 
relationship and the strength of the relationship between student context and reading 
achievement of economically disadvantaged students. 

Measures for Ethical Protection 

Many measures were taken by the researcher to ensure the rights of the 
participants were protected. First, no data was collected until the study was approved by 
the Institutional Review Board (IRB) at Columbus State University (Appendix F). The 
researcher received approval from the school system (Appendix E) before research 
began. Teacher participants were voluntary and a written consent was collected before 
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the research began (Appendix C). Teacher participants, the school principal, and the 
school system received a copy of the questionnaire and the observation tool the 
researcher used to collect teacher data. The researcher infonned participants of the 
purpose of the research and explained that there were no foreseeable risks involved. To 
ensure a safe and productive setting where participants felt they could be truthful, data 
collected from teachers was confidential. In addition, teacher names and student names 
were coded after the teacher questionnaire and the classroom observations were 
completed to safeguard identity. The data collected will be destroyed after successful 
completion of the dissertation defense. 


44 



CHAPTER 4 


Data Analysis 


Results 

This study examined the higher-order thinking question levels employed by third- 
grade and fifth-grade teachers during reading instruction and the student context when the 
HOT questions were asked. The targeted group of students for this study was the 
students meeting eligibility requirements for free/reduced-price meals. The purpose of 
this study was to determine the impact of the teachers’ HOT questions to define if the 
HOT question levels and student context affected the reading achievement of 
economically disadvantaged students as measured by the 2012 reading CRCT scores and 
the Literary Comprehension and Reading Skills/Vocabulary Acquisition domains. The 
study used a teacher questionnaire and researcher observation to collect the HOT levels 
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teachers use within their classrooms and the student context when HOT questions are 
asked. In addition, the researcher compared the teacher-reported questionnaire data 
surrounding use of HOT questions to the researcher’s observed teacher use of HOT 
questions. Another purpose of this analysis was to identify if teachers’ acknowledged use 
of HOT questions during reading workshop instruction was correlated to the actual 
teaching practices occurring within the classroom. 

Research question 1. The results of the study’s first question, “To what 
extent are teachers cognizant of using HOT questions during reading instruction?”, data 
were analyzed using the teachers’ questionnaire data and the researcher’s observation 
data. Table 10 displays the teacher-questionnaire data indicating the number of HOT 
questions third-grade and fifth-grade teachers reported asking during a reading workshop 
lesson. Questionnaire response choices included never or hardly ever (0-1), sometimes 
(2-3), often (4-5), and always or almost always (6 or more). Third-grade teachers 
reported they used evaluating and creating questions, the highest HOT question levels, 
most frequently (M= 3.83, SD = .37). Fifth-grade teachers reported using remembering 
and understanding questions, the lowest HOT question level, the most frequently (M = 
3.0, SD = .00). 


Table 10 

Descriptive Statistics of HOT Questions Teachers Reported Using During a Reading 
Workshop Lesson ( a n = 4, h n = 4) _ 

Third-Grade Teachers 3 Fifth-Grade Teachers' 3 

Standard Standard 

Hot question level Mean Deviation Mean Deviation 

Remembering 2.70 .73 3.00 .00 
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Third-Grade Teachers 3 Fifth-Grade Teachers' 3 


Understanding 

3.20 

.70 

3.00 

.00 

Applying 

3.50 

.51 

2.81 

.40 

Analyzing 

3.65 

.49 

2.69 

.47 

Evaluating 

3.85 

.37 

2.50 

.51 

Creating 

3.85 

.37 

1.88 

.71 


The researcher’s observation of teacher-use of HOT questions is presented in 
Table 11. The third-grade teachers were observed using remembering (M= 16.5, SD = 
3.79) and understanding questions the most frequently during reading workshop. In fifth 
grade, teachers used remembering (M= 16, SD = 4.79) and applying questions (M= 10, 
SD = 5.48) most frequently. In both third grade and fifth grade, creating questions were 
observed the least (M= 1.25, SD = .50; M= 3.75, SD = 4.34). 


Table 11 


Descriptive Statistics of HOT Questions Researcher Observed During a Reading 
Workshop Lesson ( 11 n = 4, h n = 4) 


Hot question level 

Third-Grade Teachers 3 

Fifth-Grade Teachers' 3 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Remembering 

16.50 

3.79 

16 

4.97 

Understanding 

17.75 

4.57 

9 

3.16 

Applying 

5.25 

5.19 

10 

5.48 

Analyzing 

7.75 

4.19 

8.75 

4.57 

Evaluating 

4.75 

6.18 

9 

7.75 

Creating 

1.25 

.50 

3.75 

4.34 


Intraclass correlation coefficient analysis revealed no statistically significant 


relationships between teacher-questionnaire data and researcher’s observed data (Table 
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12) in third grade. However, there was a positive, fair correlation for the HOT question 
level of evaluating {ICC = .41) and a fair positive correlation for the HOT level of 
applying {ICC = .24). In fifth grade, there were no statistically significant relationships 
in any of the HOT question levels and poor correlation of teacher reported data to 
researcher observations (Table 13). 


Table 12 

Intraclass Correlation Coefficient of 3' d Grade Teacher-Reported Use of HOT Questions 
to Researcher-Observed (n = 20) 


HOT Level 

Intraclass 

Correlation 

95% Cl 

F value 

(df) 

Sig 




1.25 


Remembering 

.01 

(.03, .10) 

(19, 19) 

.921 

.31 

Understanding 

-.01 

(-.05, .09) 

(19, 19) 

1.31 

.57 

Applying 

.24 

(-.89, .69) 

(19, 19) 

1.20 

.28 

Analyzing 

.08 

(-.29, .47) 

(19, 19) 

.72 

.35 

Evaluating 

.41 

(2.99, .46) 

AQ 

(19, 19) 

.76 
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Intraclass F value 

HOT Level Correlation 95% Cl(df)Sig 


Creating 3 

Note. 3 Data is not available as the variable is a constant. 


Table 13 

Intraclass Correlation Coefficient of 5 th Grade Teacher-Reported Use of HOT Questions 
to Researcher-Obsewed (n = 26) 


HOT Level 

Intraclass 

Correlation 

95% Cl 

F value 

(df) 

Sig 

Remembering 

.00 

(-.07, .12) 

1.00 
(25, 25) 

.50 

Understanding 

.00 

(-.09,.16) 

1.00 
(25, 25) 

.50 

Applying 

-.09 

(-.42, .28) 

.78 

(25, 25) 

.72 

Analyzing 

-.11 

(-.30, .24) 

.59 

(25, 25) 

.91 

Evaluating 

-.16 

(-.76, .35) 

.78 

(25, 25) 

.73 

Creating 

-.05 

(-.97, .49) 

.95 

(25, 25) 

.55 


Research question 2. Examining how the HOT question levels influenced 

reading CRCT scores of third and fifth-grade economically disadvantaged students 

(Table 14), the findings indicated that 2% of the variance in reading CRCT scores was 

attributed to the HOT question level of remembering ( R 2 = .02, F /44 = . 86 , p = .39). The 

predictive power added to the model by including the variable of understanding increased 

to 21%. This change was statistically significant (R 2 = .21, F 2 ,43 = 5.73 ,p = .04). Adding 

the remaining variables of applying, analyzing, evaluating, and creating explained an 

additional 8 % percentage points however, this was not statistically significant in 
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predicting the economically disadvantaged reading CRCT scores of third and fifth-grade 
students. 


Table 14 

Summary of Hierarchical Regression Analysis for HOT Questions Predicting 
Economically Disadvantaged Student Scores in Reading CRCT in 3 ld and 5th Grade (n = 
46) ' _ 


Variable 

P 

SEB 

t 

Sig 

R 2 

Remembering 

-.23 

.93 

-1.33 

.19 

.02 

Understanding 

.38 

.73 

2.18 

.04* 

.21 

Applying 

.07 

.83 

.35 

.73 

.24 

Analyzing 

.31 

1.35 

1.41 

.17 

.25 

Evaluating 

-.06 

.70 

-.29 

.78 

.26 

Creating 

-.21 

1.36 

-1.14 

.26 

.29 


*p < .05. 


Research question 3. Teacher questionnaires and researcher observations 

were used to answer the question “Does student context when teachers ask higher-order 
thinking questions effect reading achievement of third-grade and fifth-grade 
economically disadvantaged students?” Table 15 reports the number of HOT questions 
the researcher observed the third- and fifth-grade teachers asking students within varying 
context student groups (i.e. whole group, small group, and one-on-one). Third-grade 
teachers asked the most HOT questions while students were in whole group setting (M = 
29, SD = 23.96). Similarly, the fifth-grade teachers also asked the most HOT questions 
while students were in whole group (M= 31.25, SD = 7.37). 
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Table 15 


Descriptive Statistics of HOT questions Third- and Fifth-Grade Teachers Asked in Each 
Context ( a n = 4, b n = 4) _ 


Participants 

Whole group 
Mean 
(SD) 

Small group 
Mean 
(SD) 

One-on-one 

Mean 

(SD) 

3 rd grade 

29.00 

7.50 

16.75 

teachers 3 

(23.96) 

(6.24) 

(21.75) 

5 th grade 

31.25 

13.00 

12.25 

teachers’ 5 

(7.37) 

(13.04) 

(16.13) 


Note. Data was collected by the researcher during two 
thirty-minute observations during reading workshop instruction. 


A Pearson product-moment correlation coefficient was used to assess the 
relationship between the context of students (i.e. whole group, small group, and one-on- 
one) and third-grade economically disadvantaged students’ reading CRCT scores (Table 
16). There was a strong positive correlation and statistically significant effect of whole- 
group instruction and reading CRCT scores (r = .50, p = .03). Small group instruction 
had a moderate positive relationship (r = .31, p = .19) although not statistically 
significant. One-on-one context had a strong negative correlation (r = -.46, p = .04) and a 
statistically significant effect on the reading CRCT scores. 

Whole group context (r = .48,/? = .03) had a strong positive relationship and a 
statistically significant effect on the domain of Literary Comprehension scores of 
economically disadvantaged students reading achievement. Small group context (r = .40, 
p = .09) had a strong positive relationship on the domain of Literary Comprehension 
although not statically significant. A strong negative correlation and a statistically 
significant effect was reported for one-on-one context and Literary Comprehension (r = - 
.44, p = .05). 
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Whole group context (r = .33, p = .15) had a moderate positive relationship on 
Reading Vocabulary and Acquisition scores; although not statistically significant. Small 
group had no relationship to the domain scores while one-on-one group context had a 
weak negative relationship (r = -25, p = .29) to the economically disadvantaged students’ 
scores. 

Examining the scores of fifth-grade economically disadvantaged students, no 
statistically-significant results were reported on reading CRCT scores, Literary 
Comprehension, and Reading Skills and Vocabulary Acquisition in each of the contexts. 
A variation of negative relationships were reported in each of the context groups on each 
of the reading tests with scores ranging from r = -.02 to r = -.99. The strongest negative 
relationship was between small group context and reading CRCT scores (r = -.99) 
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Table 16 


Pearson r Correlation of Student Context When Teachers Ask HOT Questions to Reading 
CRCT scores, Literarv Comprehension Scores, and Reading Vocabulary and Acquisition 
( a n = 20, h n = 26) 




3 rd grade 3 



5 th grade* 3 



Whole 

Small 

One-on- 

Whole 

Small 

One-on- 

Test 

group 

Group 

One 

group 

Group 

One 

Reading CRCT 

r 

.50 

.31 

-.46 

-.13 

-.99 

-.08 

p 

.03* 

.19 

.04* 

.53 

.66 

.70 

Literary 

Comprehension 

r 

.48 

.40 

-.44 

-.02 

-.27 

-.19 

p 

.03* 

.09 

.05* 

.94 

.18 

.35 

Reading 

Vocabulary 

and 







Acquisition 

r 

.33 

.15 

-.25 

-.03 

-.11 

-.10 

p 

.15 

.53 

.29 

.88 

.60 

.63 


*p< .05. 
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CHAPTER 5 


Summary of Findings 


Overview of the study 

Duffy-Hester (1999) suggests the most important factor in students learning to 
read is the teacher. As teachers face the challenge to ensure all students can read, 
higher-order thinking skills are critical for making meaning from what is read. The 
introduction of the English Language Arts Common Core Standards document (National 
Governors Association Center for Best Practices, 2010) suggests that in order for students 
to be college and career ready, students must master the standards in the Common Core 
State Standards for reading. Many of the standards in this document require students to 
utilize HOT skills such as applying, analyzing, evaluating and creating (National 
Governors Association Center for Best Practices, 2010). 

Although many researchers have investigated the effect of HOT questions on 
students (Anderson & Krathwohl, 2001; Bloom, 1956; Knapp et ah, 1995; Pressley, 
Ranking, & Yokoi, 1996) this research was focused on the level of HOT questions and 
the student context affecting the economically disadvantaged students reading 
achievement as measured by the reading CRCT scores. Additionally, this research 
examined the correlation between teacher-reported use of HOT questions and researcher- 
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observed use of HOT questions. In this study, the researcher used two methods to collect 
data surrounding the teachers’ use of HOT questions and student context. First, the third- 
and fifth-grade teachers completed a questionnaire indicating their use of HOT questions 
during whole group, small group, and one-on-one reading instruction. Next the 
researcher conducted two thirty-minute classroom observations to record the HOT 
question levels used by third- and fifth-grade teachers and the student context when 
teachers asked the HOT questions. The dependent variable in this research study was the 
2011-2012 reading CRCT scores of the third- and fifth-grade economically 
disadvantaged students. In order to further analyze the effect of HOT questions on 
economically disadvantaged students, the researcher further investigated the effect of 
context upon two of the domains included in the reading CRCT scores (i.e. Literary 
Comprehension and Reading Skills/Vocabulary). 

Teachers' cognizant use of HOT questions. No Child Left Behind 
(NCLB, 2001) required states to make high quality professional development available to 
teachers. The Teaching Commission (2004) proposed “ongoing and targeted professional 
development to help meet the demanding standards”. Although both the initiatives 
promote teacher professional learning, details of the content and character of the 
professional learning are not defined. In this study, the teacher questionnaire asked 
teachers to report the topics of their attended professional learning within the past two 
years. All the teachers in this study reported having participated in professional learning 
or in-service during the past two years focused on best teaching practices in the area of 
HOT questioning. The researcher would hypothesize that because teachers had attended 
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professional learning on HOT questions, the teachers would use the higher-levels 
questions during classroom instruction. 

When asked to report the frequency of use of HOT questions, third-grade 
teachers’ reported using the two highest levels of HOT, evaluating and creating, most 
frequently within reading instruction. However, researcher-observations of teachers’ use 
of HOT questions indicated the lowest two levels of HOT, remembering and 
understanding, were most frequently used by teachers. A possible explanation for the 
difference may include the use of self-reported data. Cook and Campbell (1979) suggest 
participants tend to report what they believe the researcher expects to see, or report what 
reflects positively on their own abilities, knowledge, beliefs, or opinions. Because the 
teachers were aware of the researchers’ focus on HOT questions and also the school 
districts’ emphasis on teachers’ use of HOT questions, the teachers may have reported 
frequent use of the questioning levels of evaluating and creating. 

Correlation data suggested the HOT levels of applying and evaluating had a 
positive fair correlation, although not statistically significant. In this study, the third- 
grade teachers had a good understanding of how frequently they asked applying questions 
requiring students to use infonnation to solve a given problem. A possible explanation 
may be that teachers are planning for and requiring students to “apply” what is taught to 
demonstrate the students’ understanding. As the students apply what the teacher is 
presenting, the teacher is evaluating the students’ application to make mid-lesson changes 
in instruction to either provide more support to the student or to continue to build upon 
the students’ demonstrated knowledge (Teaching Excellence in Adult Literacy, 2012). 
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The findings of this research agree with Pressley, Rankin, and Yokoi (1996) and Knapp 
et al. (1995) that effective literacy teachers plan and teach both lower-order and higher- 
order skills. Third-grade teachers in this study were cognizant of how frequently they 
asked students to apply what was taught and the frequency they asked students to use 
evaluation to demonstrate their learning. 

The other levels of remembering, understanding, and analyzing presented no 
relationship between what the teacher reported and what the researcher observed. One 
possible explanation for this may be teachers often become focused on the content they 
are presenting to students. This focus on content may overshadow the teachers’ 
cognizance of the level of the questions presented to the students. 

Three of the four teachers in fifth grade reported having less than four hours of 
professional learning or in-service focused on best teaching practices in the area of HOT. 
Teachers reported using the highest two levels of HOT, evaluating and creating, the least 
which was consistent with the researcher’s observations. Although contrary to research 
suggesting the higher-level questions have the greatest impact on student achievement 
(Redfield & Rousseau, 1981) the test scores of the fifth-grade economically 
disadvantaged students for the past two years indicate a 99 to 100 percent passing score 
for economically disadvantaged students (Table 8). This suggests although the fifth- 
grade teachers are using the highest level of HOT questions the least, the economically 
disadvantaged students continued to meet or exceed the standards. 

Correlational data presented no statistically-significant differences in the question 
levels fifth-grade teachers reported and the researcher observed data. In addition, there 
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was no relationship or a negative negligible relationship between teacher reported and 
researcher observed data. Topping & Ferguson (2005) researched highly-effective 
literacy teachers to discover if teacher perceptions of their teaching behaviors 
corresponded with researchers’ observation of teachers’ behaviors. They observed 
teachers during shared reading and general reading sessions concluding that one of the 
top two effective strategies teachers use with students was questioning. Their 
comparison of teacher reported data to their observations indicated the teachers were 
unaware of some of the behaviors they employed during reading instruction (often 
reporting less complex behaviors than the researcher observed). This research found 
similar results when the fifth-grade teachers frequently reported the opposite of the 
researcher’s observations. A possible reason for the differences may be the human 
memory is fallible (Schacter, 1999). Previous researchers (Greenwald, 1980; Newman 
& Baumeister, 1996; Ross, 1989) suggest people misremember their past in order to 
confirm their self-theories and serve their current needs and motives. This may explain 
why the teachers reported using the higher-level questions rather than the lower-level 
questions the researcher observed. 

Comparable to the fifth- grade teachers in this study, the third-grade teachers’ 
self-reported data agreed with the findings of Topping and Ferguson (2005). Although 
the teachers reported they used evaluating and creating most frequently, observations 
suggested they asked remembering and understanding questions most frequently. Again, 
confirming the results of the Topping and Ferguson (2005) study. 
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The suggested findings of this research imply some teachers believe they are 
using the higher-level HOT questions frequently during reading instruction. However, 
when providing actual reading instruction, the teachers frequently ask more lower- 
ordering thinking questions than higher-order thinking questions. This finding suggests 
teachers are unaware of the levels they ask students and may need further professional 
development of how to incorporate the higher-level HOT questions within reading 
instruction. Additionally, focused lesson plans incorporating higher-level questions and 
student activities within reading instruction may assist teachers’ awareness and use of 
HOT questions within the instruction. 

HOT question levels. The results of the hierarchical regression analysis of 

the teachers’ use of the various HOT level questions effect on economically 
disadvantaged students reading achievement suggest only the level of remembering and 
understanding combined have a statistically significant effect on the reading CRCT 
scores. The addition of the four higher-level questions levels (applying, analyzing, 
evaluating, and creating), did not have a statistically significant effect on the 
economically disadvantaged students’ reading CRCT scores. Although Flynn (2007), in 
his research of effective teachers, suggested high-quality questioning led to increased 
reading skills for economically disadvantaged students, this study did not confirm his 
findings. 

One possible explanation for this discrepancy may be the format of the reading 
CRCT assessment. The 2011-2012 CRCT reading assessment was a multiple-choice test 
which may possibly limit how the HOT thinking skills can be assessed. However, the 
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Common Core State Standard assessments being developed by the Partnership for 
Assessment of Readiness for College and Careers (PARCC) (2012) are purposed to 
assess and measure higher-order skills beginning in 2013-2014. The addition of 
constructed response questions and student writing to assess reading comprehension will 
require students to use the higher-level skills such as synthesize, evaluate, and create 
making the teachers’ use of HOT question even more relevant to student success. 

This research agrees with the Bitter, O’Day, Gubbins, and Socias (2009) 
replicated study in 2005-2006 of high-level questioning and growth in student reading 
achievement. When these researchers replicated their study from the previous year, they 
did not find the same significant positive effects of high-level questions. When they 
explored possible reasons for the discrepancies, they discovered an increase in the 
number of English Learners (EL). They suggested different literacy practices for 
different groups of students need further research. In this research that focused on the 
student group of economically-disadvantaged, the teachers’ use of HOT questions did not 
predict student reading achievement. The findings of this study agree with Bitter et al. 
(2009) as varying groups (i.e. economically disadvantaged students) may be affected 
differently by the HOT question levels. 

Student context. The researcher collected data to determine if student 

context (i.e. whole group, small group, or one-on-one) when teachers’ asked HOT 
questions affected the reading achievement of economically disadvantaged students. In 
third grade, whole group had a statistically significant effect and positive correlation on 
the reading CRCT scores and the domain scores of Reading Vocabulary and Acquisition. 
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Small group did not have a statistically significant effect on any of the economically 
disadvantaged students reading scores. One-on-one context had a strong, negative 
correlation and a statistically significant effect on reading CRCT and Literary 
Comprehension scores. This data suggests HOT questions asked during whole-group 
instruction have the most positive effect on third-grade economically disadvantaged 
students’ reading achievement. A possible explanation for this positive effect may be all 
students have the opportunity to hear the teacher’s questions and also to hear the 
students’ responses. The teacher’s immediate feedback to the student’s response will 
also provide either further explanation to clear up any student misunderstandings or 
confirm the accuracy of the student’s response. Other researchers (Van den Brand en, 
2000; Morrow& Gambrell, 2000) suggest diverse learners’ reading comprehension is 
increased when the meaning of a text is constructed collaboratively. The practice of 
constructing meaning in a whole group may explain why the whole-group questioning 
positively affected the economically disadvantaged students reading. 

A possible explanation for the negative correlation of one-on-one and teacher 
HOT questioning may be the influence of teacher expectation on the student (Hattie, 
2009). The teacher may have lower expectations when working one-on-one with a 
student. Hattie (2009) suggests that effective teachers not only use particular teaching 
methods, but also have high expectations for all students and create positive student- 
teacher relations. 

In fifth-grade, the student context when teachers asked HOT questions did not 
have a statistically significant effect on the economically disadvantaged students’ reading 
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achievement. Negative correlation across all of the reading assessments for economically 
disadvantaged students was presented. One possible explanation may be the students 
have developed personal learning strategies and their ability to leam does not depend on 
the context. However, this may be an area for further research. 

McNeil (2010) suggested upon completion of his research surrounding HOT 
questions and English-Language Learners (ELL), further research should be focused on 
student context in which teachers ask HOT questions. This research study of context 
compared whole group, small group, and one-on-one, found whole group questioning to 
have the most positive effect on economically disadvantaged students’ reading 
achievement in third grade. However, student context in fifth-grade did not have a 
statistically significant effect on the reading achievement of economically disadvantaged 
students. 

King, Goodson, and Rohani (1998) suggest small groups can be effective in the 
development of students’ thinking skills. Although this may be true in a variety of 
classrooms, in this research, teachers asking HOT questions to small groups did not have 
a statistically significant affect in either third- or fifth-grade economically disadvantaged 
student reading achievement. Lurther details of how King, Goodson, and Rohani (1998) 
measured the effectiveness of students’ thinking skills may provide more information to 
support their suggestions. 
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Implications of the study 

Centered on the review of literature and the findings of this study, the following 
implications for teachers’ use of HOT questions with economically disadvantaged 
students are worth further consideration. 

• Analysis of the teachers’ use of HOT question levels suggests the questioning 
levels of remembering and understanding together have a statistically significant 
effect on the reading CRCT scores of the economically disadvantaged students. 
Although all the HOT levels together did not have a statistically significant effect, 
there was an 8% point increase from the low level of understanding to the highest 
level of creating. This suggests the higher levels of questioning do not have a 
statistically significant effect on the reading CRCT scores of economically 
disadvantaged students 

• The reading CRCT scores and the Literary Comprehension scores of third-grade 
economically disadvantaged students in whole group settings suggest statistically 
significant results. Whole group had a moderate, positive correlation on the other 
domain considered, Reading Vocabulary and Acquisition, although not 
statistically significant {p = .15). A recommendation for third-grade teachers is to 
frequently use whole group settings to affect the third-grade economically 
disadvantaged students’ reading achievement. 

• The reading CRCT scores and the domain scores of fifth-grade economically 
disadvantaged students were not statistically significantly affected by the context. 
In addition, each context had a negative relationship on the scores. Small group 
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had a very strong negative relationship (r = -.99) with the reading CRCT scores 
suggesting that small group instruction should be used sparingly with fifth-grade 
economically disadvantaged students. 

• Third- and fifth-grade teachers frequently reported the opposite of what the 
researcher observed. A possible risk-free strategy for teachers to become more 
self-aware may be to ask a colleague to observe and critique the teachers’ use of 
higher-order thinking questions. Robbins (1991) suggests peer coaching is a way 
to increase feedback about instruction and curriculum, expand, refine, and build 
new skills. 


Recommendations for further study 

Based upon the findings of this study, the following recommendations are made for 
further research: 

1. An area for further investigation may be the teachers’ use of HOT questions 
during one-on-one instruction (Table 16) in third grade. The results of this study 
suggested a negative correlation between one-on-one context and economically 
disadvantaged student reading scores. Research focused on effective teaching 
practices with economically disadvantaged students during one-on-one instruction 
may lead to increased learning for these students. For example, instead of 
teachers asking HOT questions to economically disadvantaged students during 
one-on-one instruction, it may be more beneficial to students if teachers focus on 
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vocabulary strategies to expand the economically disadvantaged students’ 
vocabulary repertoire. 

2. This research examined data surrounding reading workshop and HOT question 
levels. Examining the use of HOT questions in other content area may find 
varying results. 

3. In fifth-grade, the student context when teachers asked HOT questions did not 
have a statistically significant effect or a strong correlation across any of the 
reading assessments for economically disadvantaged students. This may be an 
area for further research to identify possible causality and to identify if HOT 
questions used in other content areas (i.e. Math or Science) are affected by student 
context in fifth-grade. 

4. The use of teacher-reported data and the researcher’s observation data was 
implemented within this study. Teacher interviews may provide more in-depth 
understanding as to the explanations for the teachers’ choice of leveled questions. 

Limitations of the Study 

The limitations of this study included the use of self-reported data collected by the 
teachers’ questionnaire and voluntary participation of the teachers. In addition, the 
researcher examined data of one school year (2011-2012) from one southern rural school 
of 4 third-grade and 4 fifth-grade teachers and 46 third-grade and fifth-grade 
economically disadvantaged students. Although scores from a criterion-referenced test 
were used to measure student success, other student successes such as increases in 
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gradient text level or increase in reading fluency, were not examined. Student reading 
skills such as phonemic and phonological awareness, sight word knowledge, and 
vocabulary were also not measured with this research. 

An additional limitation was the comparison of the previous year’s reading CRCT 
scores of the third-grade students. This limitation was present because this year’s third- 
grade students did not take a criterion-referenced competency test in second grade. 
Therefore, a comparison of previous scores to current scores is unavailable. 
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APPENDICES 


Appendix A 


Higher-Order Thinking Questions 


Teacher Questionnaire 


Name 

Date 


This research project will investigate the relationship between economically 
disadvantaged students’ reading achievement and teachers’ use of higher-order thinking 
questions. 

This questionnaire is designed to gather infonnation regarding teachers’ use of 
higher level questions during reading instruction in an effort to improve reading 
instruction of economically disadvantaged students. It is addressed to teachers of third- 
grade and fifth-grade students, who are asked to supply information about their use of 
higher-order thinking questions. 

Your participation is voluntary. I realize that you are very busy, but urge you to 
complete this questionnaire as accurately as possible. The information you provide is 
being collected for research purposes only and will be kept strictly confidential. The 
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questionnaire should take approximately ten minutes to complete. I appreciate the time 
and effort that this takes and thank you for your cooperation and your contribution. 


Please answer questions directly on this questionnaire. Please return the questionnaire to 
Ruth Bennett by May 1, 2012 in the accompanying envelope. Thank you! 

1. What grade students do you currently teach? 

o 3 rd grade 
o 5 th grade 

2. Have you changed your methods of teaching reading in the past two years? 

o Yes 
o No 

i. If so, how have you changed your reading instruction: 


3. Consider all of the professional development activities in which you participated 
during the last two years. To what extent did you leam about each of the 
following topics? Check one box on each line. 


Professional Development Topic 

(Number of Professional Learning 
hours) 

Not at all 

(0 hours) 

Small 

Extent 

(1-2 hrs.) 

Moder¬ 

ate 

Extent 
(3-4 hrs.) 

Large 

Extent 

(5 or more 
hrs.) 

How students leam to read 





Content standards in reading 
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Professional Development Topic 

(Number of Professional Learning 
hours) 

Not at all 

(0 hours) 

Small 

Extent 

(1-2 hrs.) 

Moder¬ 

ate 

Extent 
(3-4 hrs.) 

Large 

Extent 

(5 or more 
hrs.) 

Instructional methods for teaching 
reading 





Strategies for teaching reading to 
students from diverse backgrounds 
(including English Language 

Learners) 





Best teaching practices in the area of 
Higher Order Thinking Skills 






Classroom Organization and Reading Instruction 


1. In a typical school week, what percentage of your reading instruction do you 
devote to the following? 

o Teaching the class as a whole group _% 

o Working with a small group (4-6 students) _% 

o Working with individual students _% 

Total = 100% 


2. During a reading lesson with your students in a whole group setting, how often 
do you ask your students to do the following? Check one box on each line. 



Never or 
hardly ever 

Sometimes 

Often 

Always or 
Almost 
Always 

(Number of times per lesson) 

(0-1) 

(2-3) 

(4-5) 

(6 or 
more) 

Remember, identify, quote, define 





Understand, explain, interpret 
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(Number of times per lesson) 

Never or 
hardly ever 

(0-1) 

Sometimes 

(2-3) 

Often 

(4-5) 

Always or 
Almost 
Always 

(6 or 
more) 






Apply solve, use concepts in a new 
situation 





Analyze, compare, classify 





Evaluate, assess, recommend, convince 





Create, modify, generalize, integrate 






3. During a reading lesson with your students in a small group setting (4-6 
students), how often do you ask students to do the following? Check one box on 


each line. 


(Number of times per lesson) 

Never or 
hardly ever 

(0-1) 

Sometimes 

(2-3) 

Often 

(4-5) 

Always or 
Almost 
Always 

(6 or 
more) 

Remember, identify, quote, define 





Understand, explain, interpret 





Apply solve, use concepts in a new 
situation 





Analyze, compare, classify 





Evaluate, assess, recommend, convince 
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Never or 
hardly ever 

Sometimes 

Often 

Always or 
Almost 
Always 

(Number of times per lesson) 

(0-1) 

(2-3) 

(4-5) 

(6 or 
more) 

Create, modify, generalize, integrate 






4. During a reading conference with a student reading independently, how often do 


you ask your student to do the following? Check one box on each line. 



Never or 
hardly ever 

Sometimes 

Often 

Always or 
Almost 
Always 

(Number of times per lesson) 

(0-1) 

(2-3) 

(4-5) 

(6 or 
more) 

Remember, identify, quote, define 





Understand, explain, interpret 





Apply solve, use concepts in a new 
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Never or 
hardly ever 

Sometimes 

Often 

Always or 
Almost 
Always 

(Number of times per lesson) 

(0-1) 

(2-3) 

(4-5) 

(6 or 
more) 

situation 





Analyze, compare, classify 





Evaluate, assess, recommend, convince 





Create, modify, generalize, integrate 






5. To what extent do you consider the Bloom’s Taxonomy question level before 
asking students questions during reading instruction? 
o Not at all 
o Small extent 
o Moderate 
o Large 


Please include any additional comments you have about your use of higher-order thinking 
skills in your classroom. 
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Thank you for completing this questionnaire. 

Please return it to Ruth Bennett in the self-addressed envelope by May 1, 2012. 


Adapted from Mullis, I. V., Martin, M. O., Gonzalez, E. J., Kennedy, A. M, & Foy, P. 
(2007). PIRLS 2006 International Report: IEA’s Progress in International Literacy 
Study in Primary School in 40 Countries. Chestnut Hill, MA: Boston College. 


Appendix B 


Higher-Order Thinking (HOT) Questions: Researcher Observations 

(Adapted from revised Bloom’s Taxonomy of Educational Objectives) 
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Date 


Start time 


End time 


Observer_ 

Teacher observed_ Grade _ 

Lesson observed during: (circle one) 

Mini-lesson Conferences or Strategy Groups Teaching Share 


HOT Question 

Whole group 

Small group 

One-on-one 

Creating 




Evaluating 




Analyzing 




Applying 




Understanding 




Remembering 





Student Name 

HOT Question Level 

Correct Answer + or 
Incorrect Answer - 














Notes:_ 

Anderson, L.W., & Krathwohl, D. (Eds.) (2001). A taxonomy for learning, teaching and 
assessing: a revision of Bloom's Taxonomy of Educational Objectives. New York: 
Longman. 
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Appendix C 


Informed Consent for 

"Higher-Order Thinking Questions Affecting Economically 

Disadvantaged Students' Reading Achievement" 

Dear Teacher, 

The following information is provided for you to decide whether you wish to participate 
in the present research study. Before you give your consent to volunteer, it is important 
that you read the following infonnation and ask as many questions as necessary to be 
sure you understand what you will be asked to do. 

Principal Investigator: Ruth Bennett; (Ed. D. Candidate, Columbus State 
University); Degree(s): B.A., State University of New York, 1995, M. Ed., State 
University of New York, 1999, Ed. S. Columbus State University, 2005 
Faculty Sponsor: Dr. Deirdre Greer; Assistant Department Chair of Teacher 
Education, Columbus State University, Columbus, GA 31907 

Purpose of the Research 

This research study is designed to investigate the effect of teachers’ use of higher-order 

thinking questions on economically disadvantage students’ reading achievement. A 

secondary purpose is to identify if student group structure (i.e. one-to-one, small group, 
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or whole group) when HOT questions are asked contributes to reading achievement of 
economically disadvantaged students. 

You were selected as a participant because you are a third- or fifth-grade teacher at the 
researcher’s chosen school. The researcher’s focus on third- and fifth-grade teachers and 
students is due to the fact that the reading CRCT is a “gate” for both of these grades. 

The data from this research will be used: 

1) In the fulfillment of the Ed. D. candidate dissertation research requirements 

2) In a possible academic article 

Procedures 

If you volunteer to participate in this study, you will be asked to complete a questionnaire 
collecting information about your teaching background and instructional practices as they 
relate to reading instruction. You will be asked to allow the researcher to disaggregate 
the data gathered during two thirty-minute classroom observations conducted between 
April and May 2012 to identify the level and frequency of use of higher-order thinking 
questions and the context of students during your questioning (i.e. whole group, small 
group, or one-on-one) . You will receive a copy of the final report before it is handed in, 
so that you have the opportunity to suggest changes to the researcher, if necessary. 

Potential Risks or Discomforts 

There are no foreseeable risks associated with the study. 
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Potential Benefits of the Research 


Your participation in this study may benefit you by providing insight into your own use 
of higher-order thinking questions. In addition, further details as to the influence of 
higher-order thinking questions on the reading achievement of economically 
disadvantaged students in your classroom may be revealed. 

Confidentiality and Data Storage 

Your real name will not be used at any point of infonnation collection. I will maintain 
the confidentiality of all collected data; however, you and the other third/fifth grade 
teachers will be aware that each of you is participating in the study. The raw data will be 
accessible only to me and my dissertation committee members for the purpose of analysis 
but will be protected by the rule of confidentiality. All research information will be kept 
in a locked file cabinet drawer in my home for a period of five years, after which it will 
be destroyed. 

Participation and Withdrawal 

Your participation in this research study is voluntary. As a participant you may refuse to 
participate at any time. If you decide to participate, you are free to withdraw at any time. 
Your decision about whether to participate or to withdraw your participation will in no 
way affect evaluation of your teaching now or in the future. To withdraw from the study 
please contact Ruth Bennett via letter to Ruth Bennett, 301 Robert Bryson Smith 
Parkway, Bonaire, GA 31005. If the researcher determines circumstances requiring 
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termination of your participation in the research, a certified letter explaining your 
termination in the study will be mailed to you. 

Questions about the Research 

If you have questions about the research at any time, please speak with Ruth Bennett at 
301 Robert Bryson Smith Parkway, Bonaire, GA 31005 or (478) 929-6113 ext. 30105. 
This project has been reviewed and approved by Columbus State University’s 
Institutional Review Board. If you believe there is any infringement upon your rights as 
a research subject, you may contact my chair, Dr. Deirdre Greer, at 706-507-8034 or the 
Columbus State University IRB at irb@columbusstate.edu or 

nicks_clayton@columbusstate.edu or IRB, Office of Provost, 4225 University Avenue, 
Columbus State University, Columbus, GA, 31907. 


I have been given the opportunity to ask questions and these have been answered to my 
satisfaction. My signature below indicates my voluntary agreement to participate in this 
research study. 

Please return one copy of this consent fonn to Ruth Bennett and keep one copy for your 
records. 

Signature of Research Participant Date 

Participant Name (Please Print) Date 

Signature of Person Obtaining Consent (optional) Date 
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Appendix D 


March 26, 2012 

Re: Doctoral Study Permission Request 

Dear Principal, 

I am currently a doctoral student at Columbus State University in the area of 
Educational Leadership. I am planning to conduct a research study focused on higher- 
order thinking questions teachers ask their students and the effect of the higher-order 
thinking questions on economically disadvantaged students. The anticipated research 
project start date will be April 20, 2012 with an end date of May 30, 2012. 

The purpose of my study is to gain information from the school’s third and fifth 
grade teachers’ regarding the use of higher-order thinking (HOT) questions and the 
teacher’s perceived use of HOT questions. In addition, I will be considering the student 
context (i.e. whole group, small group or one-on-one) to identify if group context when 
teachers ask HOT questions affect student reading achievement. While it is known that 
there is a correlation between teachers asking HOT questions and student achievement, 
the information gathered in this study may help our teachers focus on specific levels of 
questions having the greatest impact on economically disadvantaged students’ reading 
achievement. 

The school district’s participation and teachers’ participation in this study is 
completely voluntary. Should you grant permission for this research to occur, all 
information will be kept confidential and neither individual nor school will be identifiable 
from the final report. 
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I would greatly appreciate your approval to conduct this research project. If I can 
answer any questions regarding this request, please contact me at ruth.bennett@hcbe.net. 

Respectfully submitted, 

Ruth Bennett, 

Principal Researcher 

Enclosures: Teacher questionnaire 


Appendix E 


MEMORANDUM 

DATE: April 11, 2012 

TO: Ruth Bennett 

Elementary School 

FROM: 

Director of Professional Learning 


SUBJECT: RESEARCH APPROVAL REQUEST 


Your request to conduct research for your doctorate program at Columbus State 
University is approved. The title of your research project is “ Higher-Order 
Thinking Questions and Reading Achievement of Economically Disadvantaged 
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Students’'. The purpose of this study is to investigate the relationship between 
economically disadvantaged students’ reading achievement and teachers’ use of 
higher-order thinking questions. As I understand from your proposal, third and 

fifth grade teachers at_Elementary and _Elementary will complete a 

questionnaire. In addition, classroom instruction will be observed at_ 

Elementary with the purpose of identifying teacher's use of higher-order thinking 
questions and CRCT data will be reviewed. The timeframe for this research 
project is one year from the date of system approval. 

Thank you for submitting your research proposal, questionnaire, observation form, 
consent form, and the principal approval letters. 

Please keep in mind that you will be responsible for compiling the data for your 

research. The staff at_Elementary School and the Department of Testing 

and Infonnation Technology is unable to compile data for your research. Board 
policy also prohibits the use of system email for personal research. Please also 
remember teacher and student anonymity is of utmost priority for this research 
project. 

I have attached to this memorandum the_County Schools Requirements for 

Conducting Research. 

I wish you the best as you work toward earning your doctorate degree. Please let 
me know if I may be of any assistance to you again in the future. 


Appendix F 


IRB Approval 


IRB Response 
Date: April 27, 2012 

Protocol Title: Higher-Order Thinking Questions and Reading Achievement of 

Economically Disadvantaged Students 

Protocol Number: #12-039 

Principal Investigator(s): Ruth Bennett 

Faculty Supervisor: Dr. Deidre Greer 
Faculty email: greer_deirdre@columbusstate.edu 
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Dear Ms. Bennett, 


Members of the CSU Institutional Review Board have reviewed your research proposal 
identified above. It has been determined that the project is minimal risk and is classified 
as exempt under 45 CFR46.101(b) of the federal regulations. 

Your most recent revisions to the project, which include conducting the research at a 
different school and excluding potential identifying variables from the study, have been 
accepted. Approval is granted for the project for one (1) year from the date of this letter 
for approximately 18 subjects. 

Please note any change to the protocol must be submitted to the IRB before implementing 
this change. Any unanticipated harms to subjects or adverse events must be reported to 
the Office of Academic Affairs at (706) 568-2061. 

If you have further questions, please feel free to contact me. Good luck with the study. 


Sincerely, 

Clay Nicks, Ph.D. 

Chair, CSU Institutional Review Board 
Associate Professor 

Department of Health, Physical Education, and Exercise Science Columbus State 
University 
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